PSYC5141: Experimental Methods for Psychology

Mark Hurlstone, Lancaster University

Week 3, Lecture 2

Single-Factor Within-Participants ANOVA

Learning Objectives

• The F ratio for a within-participants ANOVA
• Between-participant variability and residual variability
• Removing individual differences from the error term
• Calculating and interpreting the F ratio for a single-factor within-participants ANOVA
• Assumptions of within-participants ANOVA
• Planned comparisons and post-hoc tests

The F Ratio For A Within-Participants ANOVA

The F Ratio: A Reminder

• The general equation for the F ratio is:

• For a between-participants ANOVA, the within-group variance provides the estimate of experimental error: 

Sources of Variability in Between-Participants ANOVA

• Recall that between-group variance contains two sources of variability
  • Treatment effects
  • Experimental error
• We can therefore express the F ratio conceptually as:

Sources of Variability in Between-Participants ANOVA

• Experimental error itself consists of:
  • Individual differences: variability arising because participants differ from one another
  • Random errors: other unexplained influences on scores
• Therefore:

The F Ratio for Within-Participants ANOVA

• In a within-participants design, each participant provides a score in every treatment condition
• This allows us to separate variability attributable to individual differences from the error term
• The F ratio can therefore be represented conceptually as:

The F Ratio for Within-Participants ANOVA

• Individual differences are removed from the error term
• The remaining error is called residual error
• Reducing the error term can increase the F ratio and make treatment effects easier to detect
• Formally, the F ratio is:

Between-Participant Variability and Residual Variability

Sources of Error Variability

When we obtain multiple scores from each participant, we can distinguish two ways in which participants differ:

1. Overall performance: participants may differ in their average performance across treatment conditions
2. Trends in performance: participants may differ in the size or direction of their changes between treatment conditions

These correspond to:

• Between-participant variability
• Residual variability

Between-Participant Variability

• Participants differ considerably in overall performance
• However, every participant improves by exactly 10 points

High between-participant variability / zero residual variability

Residual Variability

• Participants have identical mean scores
• However, their responses to the treatment conditions differ considerably

Zero between-participant variability / high residual variability

Residual Variability

The key distinction:

• Between-participant variability concerns differences in overall performance
• Residual variability concerns differences in trends across conditions

Removing Individual Differences from the Error Term

Why Remove Individual Differences?

When investigating treatment effects, we are primarily interested in whether participants' scores change systematically between conditions

• Differences in participants' overall performance are not necessarily relevant to this question
• Within-participants ANOVA allows us to estimate how much variability is attributable to these individual differences
• We can remove this component from the within-group variability, leaving residual variability

By removing individual differences from the error term, within-participants ANOVA can reduce experimental error, increasing the F ratio and improving statistical power

Removing Between-Participant Variability

• Recall that total variability in a between-participants ANOVA can be partitioned into:

• For a within-participants ANOVA, the within-group variability can be partitioned further:

• Therefore:

This is the fundamental additional step required when calculating the within-participants ANOVA

The Within-Participants F Ratio

• We can now express the F ratio in terms of mean squares:

The important change: Within-participants ANOVA uses residual variance rather than within-group variance as the denominator of the F-ratio

• \(MS_{BETWEEN}\) represents between-group variance.

• \(MS_{RESIDUAL}\) represents residual variance after accounting for individual differences.

Calculating and Interpreting the F Ratio for a Single-Factor Within-Participants ANOVA

A Within-Participants Experiment

A psychologist investigates whether misinformation warnings improve people's ability to distinguish between true and false information on social media

• Independent variable: Type of misinformation warning, with three levels:
  • \(A_1\): No warning
  • \(A_2\): Generic warning
  • \(A_3\): Specific warning
• Dependent variable: Percentage of social media posts correctly classified as true or false.
• Design: Five participants, each completing all three treatment conditions.

A Within-Participants Experiment: Raw Data

Each row contains three scores from the same participant, rather than three independent groups of participants

A Within-Participants Experiment: Raw Data

Grand total: \(\Sigma Y= 270 + 315 + 350 = 935\)
Grand mean: \(\bar Y=62.33\)

Calculating the Sums of Squares

Calculating the Between-Group Sum of Squares

Calculating the Between-Group Sum of Squares

Grand total: \(\Sigma Y= 270 + 315 + 350 = 935\)

Calculating the Between-Group Sum of Squares

\(N\) = the total number of observations across all treatment conditions

\(N_A\) = the number of observations in each treatment condition

Calculating the Between-Group Sum of Squares

Calculating the Within-Group Sum of Squares

Calculating the Within-Group Sum of Squares

Calculating the Total Sum of Squares

Calculating the Total Sum of Squares

Verifying Our Calculations

The calculations agree!

The Additional Step in Within-Participants ANOVA

• So far, we've calculated the same sums of squares used in between-participants ANOVA:

\[SS_{BETWEEN},\quad SS_{WITHIN},\quad SS_{TOTAL}\]

• However, we've not yet taken advantage of the fact that the same participants completed every treatment condition
• We can use these repeated observations to separate variability attributable to individual differences from residual variability

The key additional step:

\[ SS_{WITHIN}=SS_{SUBJECTS}+SS_{RESIDUAL} \]

\[ \boxed{SS_{RESIDUAL}=SS_{WITHIN}-SS_{SUBJECTS}} \]

Calculating the Between-Participant Sum of Squares

We've already seen that participants differ in their overall performance, irrespective of treatment condition

• We can measure this variability using the between-participant sum of squares, \(SS_{SUBJECTS}\)
• To calculate it, we need the total score obtained by each participant across all treatment conditions
• The calculation follows the same approach used for \(SS_{BETWEEN}\), except that we use participant totals rather than treatment totals

Calculating the Between-Participant Sum of Squares

Number of scores contributed by each participant

Calculating the Between-Participant Sum of Squares

Calculating the Residual Sum of Squares

Recall:

\[ SS_{WITHIN}=SS_{SUBJECTS}+SS_{RESIDUAL} \]

Rearranging:

\[ SS_{RESIDUAL}=SS_{WITHIN}-SS_{SUBJECTS} \]

Substituting:

\[ SS_{RESIDUAL}=650-260 \]

\[ \boxed{SS_{RESIDUAL}=390} \]

The important point: Of the 650 units of within-group variability, 260 are attributable to systematic differences between participants. The remaining 390 represent residual variability.

Partitioning Total Variability

We can now partition total variability into three components:

\[ SS_{TOTAL}=SS_{BETWEEN}+SS_{SUBJECTS}+SS_{RESIDUAL} \]

Substituting our values:

\[ 1,293.33=643.33+260+390 \]

Calculating the Degrees of Freedom

\[ df_{BETWEEN}=a-1 \]

\[ df_{WITHIN}=N-a \]

\[ df_{TOTAL}=N-1 \]

Where:

• \(a\) = number of treatment conditions

• \(N\) = total number of observations

Calculating the Degrees of Freedom

• For our misinformation experiment:

\[ a=3,\qquad N=15 \]

• Therefore:

\[ \begin{aligned} df_{BETWEEN}&=3-1=\boxed{2}\\[6pt] df_{WITHIN}&=15-3=\boxed{12}\\[6pt] df_{TOTAL}&=15-1=\boxed{14} \end{aligned} \]

Partitioning the Within-Group Degrees of Freedom

• Just as we partitioned the within-group sum of squares, we can also partition the within-group degrees of freedom:

\[ df_{WITHIN}=df_{SUBJECTS}+df_{RESIDUAL} \]

• The degrees of freedom associated with individual differences are:

\[ df_{SUBJECTS}=s-1 \]

• Where \(s\) is the number of participants

Partitioning the Within-Group Degrees of Freedom

• For our experiment, \(s=5\):

\[ df_{SUBJECTS}=5-1=\boxed{4} \]

• We can now calculate the residual degrees of freedom:

\[ df_{RESIDUAL}=df_{WITHIN}-df_{SUBJECTS} \]

\[ df_{RESIDUAL}=12-4=\boxed{8} \]

The important point: Of the 12 within-group degrees of freedom, four are associated with differences between participants, leaving eight for residual variability.

Calculating the Residual Degrees of Freedom

• We can also calculate the residual degrees of freedom directly:

\[ df_{RESIDUAL}=(a-1)(s-1) \]

• For our experiment:

\[ df_{RESIDUAL}=(3-1)(5-1) \]

\[ df_{RESIDUAL}=2\times4 \]

\[ \boxed{df_{RESIDUAL}=8} \]

• This gives the same result as subtracting the participant degrees of freedom from the within-group degrees of freedom

Calculating the Residual Degrees of Freedom

• We can check our calculations by confirming that the degrees of freedom sum to the total.

  \[ df_{TOTAL}=df_{BETWEEN}+df_{SUBJECTS}+df_{RESIDUAL} \]

  \[ 14=2+4+8 \]

   

   

   

The degrees of freedom and sums of squares both add up correctly

Calculating the Mean Squares

• With both the sums of squares and degrees of freedom established, our next step is straightforward:

\[ MS_{BETWEEN}=\frac{643.33}{2}=321.67 \]

\[ MS_{RESIDUAL}=\frac{390}{8}=48.75 \]

• We can then obtain the final F ratio:

​

\[ F=\frac{MS_{BETWEEN}}{MS_{RESIDUAL}} =\frac{321.67}{48.75} \approx\boxed{6.60} \]

Critical Values: F Table

\[ F_{obtained}=6.60 \]\[ F_{critical}(2,8)=4.46 \]\[ 6.60 > 4.46 \]

Decision: Reject \(H_0\)

ANOVA Table

Assumptions of Within-Participants ANOVA

Assumptions of Within-Participants ANOVA

Like between-participants ANOVA, within-participants ANOVA relies on several assumptions:

• Independence: Observations from different participants are independent. Observations from the same participant are expected to be related
• Normality: The residuals are approximately normally distributed
• Sphericity: The variances of the differences between every pair of treatment conditions are equal

The important difference: Sphericity replaces the homogeneity-of-variance assumption used in between-participants ANOVA

Understanding Sphericity

Sphericity concerns the variability of the difference scores between pairs of treatment conditions

For our misinformation experiment, we have three conditions:

• \(A_1\): No warning

• \(A_2\): Generic warning

• \(A_3\): Specific warning

There are three possible pairwise differences:

\[ A_2-A_1,\qquad A_3-A_1,\qquad A_3-A_2 \]

The assumption of sphericity requires:

\[ \boxed{ \sigma^2_{A_2-A_1} = \sigma^2_{A_3-A_1} = \sigma^2_{A_3-A_2} } \]

In other words, the population variances of these difference scores must be equal

Sphericity is automatically satisfied when there are only two treatment conditions, because there is only one set of difference scores

Example

For our hypothetical data, the sample variances are:

\[ s^2_{A_2-A_1}=117.5 \]

\[ s^2_{A_3-A_1}=17.5 \]

\[ s^2_{A_3-A_2}=157.5 \]

These differ considerably, although unequal sample variances do not by themselves establish that population sphericity has been violated

Testing the Assumption of Sphericity

Sphericity can be assessed using Mauchly's test.

• Null hypothesis: The assumption of sphericity holds

• Alternative hypothesis: The assumption of sphericity is violated

As with Shapiro–Wilk and Levene's tests, a non-significant result does not signify that the assumption is satisfied

What Happens When Sphericity Is Violated?

• Violating the assumption of sphericity can make the conventional within-participants F test too liberal, increasing the risk of a Type I error
• Fortunately, we can adjust the degrees of freedom used to evaluate the F ratio
• Two commonly used corrections are:

  • Greenhouse–Geisser correction: Adjusts the degrees of freedom to account for departures from sphericity

  • Huynh–Feldt correction: An alternative adjustment that is generally less conservative

• These corrections do not change the calculated F ratio itself. Instead, they modify the degrees of freedom used to determine statistical significance

Planned Comparisons and Post-Hoc Tests

Planned Comparisons

As with between-participants ANOVA, we can use planned comparisons to test specific hypotheses about differences between treatment conditions.

• Planned comparisons are specified before examining the results

• For comparisons between two conditions, we can use a paired-samples t-test, with an appropriate adjustment if multiple comparisons are conducted

Post-Hoc Tests

Post-hoc tests allow us to investigate which treatment conditions differ following an ANOVA.

• Unlike planned comparisons, post-hoc tests are not restricted to comparisons specified in advance

• Typically, we examine all possible pairwise comparisons between treatment conditions.

• Conducting multiple comparisons increases the risk of Type I errors, so an appropriate adjustment is required (e.g., the Bonferroni correction)

That's all folks!